Automated incident detection for vehicles

ABSTRACT

Examples described herein provide a computer-implemented method that includes receiving first data from a sensor of a vehicle. The method further includes determining, by a processing device, whether an incident external to the vehicle has occurred by processing the first data using a machine learning model. The method further includes, responsive to determining that an incident external to the vehicle has occurred, initiating recording of second data by the sensor. The method further includes, responsive to determining that an incident external to the vehicle has occurred, taking an action to control the vehicle.

INTRODUCTION

The present disclosure relates to vehicles and more particularly toautomated incident detection for vehicles.

Modern vehicles (e.g., a car, a motorcycle, a boat, or any other type ofautomobile) may be equipped with a vehicular communication system thatfacilitates different types of communication between the vehicle andother entities. For example, a vehicular communication system canprovide for vehicle-to-infrastructure (V2I), vehicle-to-vehicle (V2V),vehicle-to-pedestrian (V2P), and/or vehicle-to-grid (V2G) communication.Collectively, these may be referred to as vehicle-to-everything (V2X)communication that enables communication of information between thevehicle and any other suitable entity. Various applications (e.g., V2Xapplications) can use V2X communications to send and/or receive safetymessages, maintenance messages, vehicle status messages, and the like.

Modern vehicles can also include one or more cameras that provideback-up assistance, take images of the vehicle driver to determinedriver drowsiness or attentiveness, provide images of the road as thevehicle is traveling for collision avoidance purposes, provide structurerecognition, such as roadway signs, etc. For example, a vehicle can beequipped with multiple cameras, and images from multiple cameras(referred to as “surround view cameras”) can be used to create a“surround” or “bird's eye” view of the vehicle. Some of the cameras(referred to as “long-range cameras”) can be used to capture long-rangeimages (e.g., for object detection for collision avoidance, structurerecognition, etc.).

Such vehicles can also be equipped with sensors such as a radardevice(s), LiDAR device(s), and/or the like for performing targettracking. Target tracking includes identifying a target object andtracking the target object over time as the target object moves withrespect to the vehicle observing the target object. Images from the oneor more cameras of the vehicle can also be used for performing targettracking.

These communication protocols, cameras, and/or sensors can be useful formonitoring vehicles and the environment around the vehicles.

SUMMARY

In one exemplary embodiment, a computer-implemented method is provided.The method includes receiving first data from a sensor of a vehicle. Themethod further includes determining, by a processing device, whether anincident external to the vehicle has occurred by processing the firstdata using a machine learning model. The method further includes,responsive to determining that an incident external to the vehicle hasoccurred, initiating recording of second data by the sensor. The methodfurther includes, responsive to determining that an incident external tothe vehicle has occurred, taking an action to control the vehicle.

In addition to one or more of the features described herein, or as analternative, further embodiments of the method may include, responsiveto determining that an incident external to the vehicle has notoccurred, receiving third data from the sensor of the vehicle, anddetermining whether an incident external to the vehicle has occurred byprocessing the third data using the machine learning model.

In addition to one or more of the features described herein, or as analternative, further embodiments of the method may include that thesensor is a camera.

In addition to one or more of the features described herein, or as analternative, further embodiments of the method may include that thesensor is a microphone.

In addition to one or more of the features described herein, or as analternative, further embodiments of the method may include thatreceiving the first data from the sensor of the vehicle comprisesreceiving audio data from a microphone, and receiving video data from acamera.

In addition to one or more of the features described herein, or as analternative, further embodiments of the method may include fusing theaudio data and the video data.

In addition to one or more of the features described herein, or as analternative, further embodiments of the method may include, responsiveto initiating recording of the second data by the sensor, overlayinginformation on the second data.

In addition to one or more of the features described herein, or as analternative, further embodiments of the method may include that theinformation comprises location information associated with a location ofthe vehicle, a timestamp associated with a time of the incident, andspeed information associated with a speed of the vehicle at the time ofthe incident.

In addition to one or more of the features described herein, or as analternative, further embodiments of the method may include that themachine learning model is a federated learning model.

In addition to one or more of the features described herein, or as analternative, further embodiments of the method may include transmittingthe second data to a data store associated with a remote processingsystem.

In addition to one or more of the features described herein, or as analternative, further embodiments of the method may include, responsiveto determining that an incident external to the vehicle has occurred,issuing an alert to an operator of the vehicle.

In addition to one or more of the features described herein, or as analternative, further embodiments of the method may include, responsiveto determining that an incident external to the vehicle has occurred,issuing an alert to a third party remote from the vehicle.

In addition to one or more of the features described herein, or as analternative, further embodiments of the method may includereconstructing, at a remote processing system, a scene of the incidentbased at least in part on the second data and third-party data collectedby at least one third-party.

In another exemplary embodiment a system includes a sensor. The systemfurther includes a memory comprising computer readable instructions. Thesystem further includes a processing device for executing the computerreadable instructions, the computer readable instructions controllingthe processing device to perform operations. The operations includereceiving first data from the sensor of a vehicle. The operationsfurther include determining, by a processing device, whether an incidentexternal to the vehicle has occurred by processing the first data usinga machine learning model. The operations further include, responsive todetermining that an incident external to the vehicle has occurred,initiating recording of second data by the sensor. The operationsfurther include responsive to determining that an incident external tothe vehicle has occurred, taking an action to control the vehicle.

In addition to one or more of the features described herein, or as analternative, further embodiments of the system may include that thesensor is a camera.

In addition to one or more of the features described herein, or as analternative, further embodiments of the system may include that thesensor is a microphone.

In addition to one or more of the features described herein, or as analternative, further embodiments of the system may include thatreceiving the first data from the sensor of the vehicle comprisesreceiving audio data from a microphone, and receiving video data from acamera.

In addition to one or more of the features described herein, or as analternative, further embodiments of the system may include operationscomprising fusing the audio data and the video data.

In addition to one or more of the features described herein, or as analternative, further embodiments of the system may include that themachine learning model is a federated learning model.

In yet another exemplary embodiment a computer program product includesa computer readable storage medium having program instructions embodiedtherewith, wherein the computer readable storage medium is not atransitory signal per se, the program instructions executable by aprocessing device to cause the processing device to perform operations.The operations include receiving first data from a sensor of a vehicle.The operations further include determining, by a processing device,whether an incident external to the vehicle has occurred by processingthe first data using a machine learning model. The operations furtherinclude, responsive to determining that an incident external to thevehicle has occurred, initiating recording of second data by the sensor.The operations further include, responsive to determining that anincident external to the vehicle has occurred, taking an action tocontrol the vehicle.

The above features and advantages, and other features and advantages, ofthe disclosure are readily apparent from the following detaileddescription when taken in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, advantages, and details appear, by way of example only,in the following detailed description, the detailed descriptionreferring to the drawings in which:

FIG. 1 depicts a vehicle including sensors and a processing systemaccording to one or more embodiments described herein;

FIG. 2 depicts a block diagram of a system for automated incidentdetection for vehicles according to one or more embodiments describedherein;

FIG. 3 depicts a flow diagram of a method for processing environmentaldata for vehicles according to one or more embodiments described herein;

FIGS. 4A, 4B, and 4C depict scenarios according to one or moreembodiments described herein; and

FIG. 5 depicts a block diagram of a processing system for implementingthe techniques described herein according to an exemplary embodiment.

DETAILED DESCRIPTION

The following description is merely exemplary in nature and is notintended to limit the present disclosure, its application or uses. Itshould be understood that throughout the drawings, correspondingreference numerals indicate like or corresponding parts and features.

The technical solutions described herein provide for automated incidentdetection for vehicles. More particularly, one or more embodimentsdescribed herein provide for automatic traffic incident recording andreporting. One or more embodiments described herein provide forrecording an incident external to the vehicle, such as a trafficincident (e.g., a traffic stop by a law enforcement officer, anaccident, etc.), or any other event external to the vehicle that acts asa trigger and then taking an action, such as controlling the vehicle,and/or reporting the incident, such as to an emergency dispatcher,another vehicle, etc.

Conventional approaches to incident detection and reporting for vehiclesare insufficient. For example, incident detection and reporting islargely a manual process that requires human detection and triggering ofreporting. Consider the example of a law enforcement officer pullingover a target vehicle for a traffic stop. In such cases, an occupant ofthe target vehicle would have to manually detect that the target vehicleis being pulled over and then manually initiate recording, such as on amobile phone (e.g., a smart phone) or camera system within the targetvehicle, of the traffic stop. If it is desired to report the incident,such as to a family member, an emergency response agency, or the like,such reporting is typically performed manually, such as through a phonecall. Further, if an incident occurs to another vehicle other than thetarget vehicle, occupants of the target vehicle may be unaware of theincident (e.g., the occupants cannot see the incident).

One or more embodiments described herein address these and othershortcomings of the prior art by detecting incidents, initiatingrecording using one or more sensors (e.g., cameras, microphones, etc.),and reporting the incident. As one example, a method according to one ormore embodiments can include detecting an incident (e.g., detecting alaw enforcement vehicle), initiating recording of audio and/or video,overlaying data (e.g., speed, location, timestamp, etc.) on the video,uploading the audio and/or video recordings to a remote processingsystem (e.g., a cloud computing node of a cloud computing environment),and issuing an alert (also referred to as a “notification”). In someexamples, the audio and/or video recordings can be used to reconstruct ascene or incident. In some examples, one or more vehicles involved inthe incident (e.g., a target vehicle, the law enforcement vehicle, etc.)can be controlled, such as by causing windows to roll down, causinglights to be turned on, causing an alert within one or more of thevehicles to be issued, and the like.

One or more embodiments described herein provide advantages andimprovements over the prior art. For example, the described technicalsolutions provide video/audio of what happened when an incident occurs(e.g., when a vehicle operator was pulled over by law enforcement) andalso provides evidence of the incident in real-time, including alerts tothird parties of the incident. Further, the described technicalsolutions provide real time assistance to the vehicle operator during atraffic stop or incident by providing real time video and/or audio.Further advantages of the present techniques can include reducedbehavioral incidents and better behavior from parties involved in theincident, which can deter escalation, decrease violent approaches andoutcomes, and the like. Further advantages include using data about adetected event to control a vehicle, such as to steer the vehicle clearof an approaching emergency vehicle.

FIG. 1 depicts a vehicle 100 including sensors and a processing system110 according to one or more embodiments described herein. In theexample of FIG. 1 , the vehicle 100 includes the processing system 110,cameras 120, 121, 122, 123, cameras 130, 131, 132, 133, a radar sensor140, a LiDAR sensor 141, and a microphone 142. The vehicle 100 may be acar, truck, van, bus, motorcycle, boat, plane, or another suitablevehicle 100.

The cameras 120-123 are surround view cameras that capture imagesexternal to, and in near proximity to, the vehicle 100. The imagescaptured by the cameras 120-123 together form a surround view (sometimesreferred to as a “top-down view” or a “bird's eye view”) of the vehicle100. These images can be useful for operating the vehicle (e.g.,parking, backing, etc.). These images can also be useful for capturingan incident, such as a traffic stop, an accident, etc. The cameras130-133 are long-range cameras that capture images external to thevehicle and farther away from the vehicle 100 than the cameras 120-123.These images can be useful for object detection and avoidance, forexample. These images can also be useful for capturing an incident, suchas a traffic stop, an accident, etc. It should be appreciated that,although eight cameras 120-123 and 130-133 are shown, more or fewercameras may be implemented in various embodiments.

Captured images can be displayed on a display (not shown) to provideexternal views of the vehicle 100 to the driver/operator of the vehicle100. The captured images can be displayed as live images, still images,or some combination thereof. In some examples, the images can becombined to form a composite view, such as the surround view. In someexamples, the images captured by the cameras 120-123 and 130-133 can bestored to a data store 111 of the processing system 110 and/or to aremote data store 151 associated with a remote processing system 150.

The radar sensor 140 measures range to a target object by transmittingelectromagnetic waves and measuring the reflected waves with a sensor.This information is useful for determining a target object'sdistance/location relative to the vehicle 100. It should be appreciatedthat the radar sensor 140 can represent multiple radar sensors.

The LiDAR (light detection and ranging) sensor 141 measures distance toa target object (e.g., other vehicle 154) by illumining the target withpulsed or continuous wave laser light and measuring the reflected pulsesor continuous wave with a detector sensor. This information is usefulfor determining a target object's distance/location relative to thevehicle 100. It should be appreciated that the LiDAR sensor 141 canrepresent multiple LiDAR sensors.

The microphone 142 can record soundwaves (e.g., sounds or audio). Thisinformation is useful for recording sound information about the vehicle100 and/or the environment proximate to the vehicle 100. It should beappreciated that the microphone 142 can represent multiple microphonesand/or microphone arrays, which can be disposed in or on the vehiclesuch that the microphone 142 can record soundwaves in an interior (e.g.,passenger compartment) of the vehicle and/or external to the vehicle.

Data generated from the cameras 120-123, 130-133, the radar sensor 140,the LiDAR sensor 141, and/or the microphone 142 can be used to detectand/or track a target object relative to the vehicle 100, to detect anincident, and the like. Examples of target objects include othervehicles (e.g., the other vehicle 154), emergency vehicles, vulnerableroad users (VRUs) such as pedestrians, bicycles, animals, potholes, oilon a roadway surface, debris on a roadway surface, fog, flooding, andthe like.

The processing system 110 includes a data/communication engine 112, adecision engine 114 to detect and classify, a control engine 116, thedata store 111, and a machine learning (ML) model 118. Thedata/communication engine 112 receives/collects data, such as fromsensors (e.g., one or more of the cameras 120-123, 130-133; the radarsensor 140; the LiDAR sensor 141; the microphone 142; etc.) associatedwith the vehicle 100 and/or receives data from other sources such as theremote processing system 150 and/or the other vehicle 154. The decisionengine 114 processes the data to detect and classify incidents. Thedecision engine 114 can utilize the ML model 118 according to one ormore embodiments described herein. An example of how the decision engine114 processes the data is shown in FIG. 2 , among others, and isdescribed further herein. The control engine 116 controls the vehicle100, such as to plan a route and execute a driving maneuver (e.g.,change lanes, change velocity, etc.), initiate recording from sensors ofthe vehicle 100, cause recorded data to be stored in the data store 111and/or the data store 151 of the remote processing system 150, andperform other suitable actions. Although not shown, the processingsystem 110 can include other components, engines, modules, etc., such asa processor (e.g., a central processing unit, a graphics processingunit, a microprocessor, the CPU 521 of FIG. 5 , etc.), a memory (e.g., arandom-access memory, a read-only memory, the RAM 524 of FIG. 5 , theROM 522 of FIG. 5 , etc.), data store (e.g., a solid state drive, a harddisk drive, the hard disk, the mass storage 534 of FIG. 5 , etc.) andthe like.

The processing system 110 can be communicatively coupled to a remoteprocessing system 150, which can be an edge processing node as part ofan edge processing environment, a cloud processing node as part of acloud processing environment, or the like. The processing system 110 canalso be communicatively coupled to one or more other vehicles (e.g.,other vehicle 154). In some examples, the processing system 110 iscommunicatively coupled to the processing system 150 and/or the othervehicle 154 directly (e.g., using V2V communication), while in otherexamples, the processing system 110 is communicatively coupled to theprocessing system 150 and/or the other vehicle 154 indirectly, such asby a network 152. For example, the processing system 110 can include anetwork adapter (not shown) (see, e.g., the network adapter 526 of FIG.5 ). The network adapter enables the processing system 110 to transmitdata to and/or receive data from other sources, such as other processingsystems, data repositories, and the like including the remote processingsystem 150 and the other vehicle 154. As an example, the processingsystem 110 can transmit data to and/or receive data from the remoteprocessing system 150 directly and/or via the network 152.

The network 152 represents any one or a combination of different typesof suitable communications networks such as, for example, cablenetworks, public networks (e.g., the Internet), private networks,wireless networks, cellular networks, or any other suitable privateand/or public networks. Further, the network 152 can have any suitablecommunication range associated therewith and may include, for example,global networks (e.g., the Internet), metropolitan area networks (MANs),wide area networks (WANs), local area networks (LANs), or personal areanetworks (PANs). In addition, the network 152 can include any type ofmedium over which network traffic may be carried including, but notlimited to, coaxial cable, twisted-pair wire, optical fiber, a hybridfiber coaxial (HFC) medium, microwave terrestrial transceivers, radiofrequency communication mediums, satellite communication mediums, or anycombination thereof. According to one or more embodiments describedherein, the remote processing system 150, the other vehicle 154, and theprocessing system 110 communicate via a vehicle-to-infrastructure (V2I),vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), and/orvehicle-to-grid (V2G) communication.

The features and functionality of the components of the processingsystem 110 are now described in more detail. The processing system 110of the vehicle 100 aids in automated incident detection for vehicles.

According to one or more embodiments described herein, the processingsystem 110 combines sensor input and driving behavior with artificialintelligence (AI) and machine learning (ML) (e.g., federated learning)to determine when the vehicle 100 is involved in an incident (e.g., atraffic stop) and automatically takes action(s), such as recording datausing sensors associated with the vehicle, connecting to third parties,controlling the vehicle (e.g., rolling down windows, turning on hazardlights and/or interior lights, etc.), adding overlay information torecorded data (e.g., speed, GPS, and time stamp added to a recordedvideo), and the like. The processing system 110 can also issuenotifications/alerts, such as providing a message on a display of thevehicle 100 communicating to the operator/occupant that the incident hasoccurred, notifying emergency contacts and/or emergency dispatcher ofthe incident.

According to one or more embodiments described herein, the processingsystem 110 can perform automatic AI/ML triggering of features based onfusion of sensor data (e.g., data from cameras, microphones, etc.) withdriving behavior observed by exterior vehicle sensors (e.g., one or moreof the cameras 120-123, 130-133; the microphone 142, etc.). Theprocessing system 110 can incorporate AI/ML (for example, throughenhanced federated learning) triggers to initiate recording, such asemergency lights, sirens, speed, and vehicle harsh maneuvering. In someexamples, the processing system 110 can cause local and/or remote datacapturing/recording to ensure data ownership/security. For example, rawdata is saved locally in the vehicle 100 (e.g., in the data store 111)and in a mobile device (not shown) of an operator/occupant of thevehicle 100. Further the federated learned data and 3D reconstructionprimitives can be uploaded to third parties, such as the remoteprocessing system 150 and/or the other vehicle 154.

The processing system 110 can also, in some examples, enable third-partydata collection and/or notifications and can provide for multi-vehicleobservation/processing. For example, the processing system 110 can sendan alert to an emergency dispatch service (e.g., the remote processingsystem 150) to initiate emergency operations. This can include sendingthe data collected by vehicle sensors to the remote processing system150 in some examples. The processing system 110 can also send an alertto the other vehicle 154 to cause the other vehicle (e.g., a third-partywitness) to collect data using one or more sensors (not shown)associated with the other vehicle 154. In some examples, the processingsystem 110 can access data from the other vehicle 154 (which mayrepresent one or more vehicles) for 3D scene reconstruction throughfederated learning (or other suitable machine learning technique). Forexample, a multiple view/camera scene reconstruction techniques can beimplemented using video collected from the vehicle 100 (and one or moreneighboring vehicles (e.g., the other vehicle 154)) for the 3D scenemodeling, and the audio associated with the video can be saved orenhanced by noise cancelling techniques. When the data from theneighboring vehicle (e.g., the other vehicle 154) within a proximity tothe vehicle 100 are processed for the 3D scene reconstruction in thevehicle 100 or a cloud computing node (e.g., using either motion stereoor shape-from motion), the vehicle 100 sends relevant data/model usingmachine learning or federated learning approaches to protect the dataprivacy. That is, data that are not deemed relevant (e.g., datacollected from before the incident, data collected from the passengercompartment of a neighboring vehicle, etc.) are not sent for scenereconstruction to provide for data privacy.

According to one or more embodiments described herein, the processingsystem 110 can, upon a detection of a law enforcement vehicle forexample, prepare the vehicle 100, such as by rolling up/down windows,turning on hazard lights, turning on interior lights, providing amessage on a display of the vehicle (e.g., “emergency vehicle behind”),and the like.

Turning now to FIG. 2 , a block diagram of a system 200 for automatedincident detection for vehicles is provided according to one or moreembodiments described herein. In this example, the system 200 includesthe vehicle 100, the remote processing system 150, and the other vehicle154 communicatively coupled by the network 152. It should be appreciatedthe other vehicle 154 can be configured similarly to the vehicle 100 asshown in FIG. 1 and as described herein. In some examples, additionalother vehicles can be implemented.

As in FIG. 1 , the vehicle 100 includes sensors 202 (e.g., one or moreof the cameras 120-123, 130-133, the radar sensor 140, the LiDAR sensor141, the microphone 142, etc.). At block 204, the data/communicationengine 112 receives data from the sensors 202. This can include activelycollecting the data (e.g., causing the sensors 202 to collect the data)or passively receiving the data from the sensors 202.

The decision engine 114 processes the data collected at block 204 by thedata/communication engine 112. Particularly, at block 206, the decisionengine 114 monitors the sensors 202, using the data received/collectedat block 204, for an indication of an incident. According to one or moreembodiments described herein, the decision engine 114 can utilizeartificial intelligence (e.g., machine learning) to detect featureswithin the sensor data (e.g., a captured image, a recorded soundwave,etc.) that are indicative of an incident. For example, features commonlyassociated with an emergency vehicle can be detected, such as flashinglights, sirens, indicium/symbol on the vehicle, etc.

More particularly, aspects of the present disclosure can utilize machinelearning functionality to accomplish the various operations describedherein. More specifically, one or more embodiments described herein canincorporate and utilize rule-based decision making and artificialintelligent (AI) reasoning to accomplish the various operationsdescribed herein. The phrase “machine learning” broadly describes afunction of electronic systems that learn from data. A machine learningsystem, module, or engine (e.g., the decision engine 114) can include atrainable machine learning algorithm that can be trained, such as in anexternal cloud environment, to learn functional relationships betweeninputs and outputs that are currently unknown, and the resulting model(e.g., the ML model 118) can be used to determine whether an incidenthas occurred. In one or more embodiments, machine learning functionalitycan be implemented using an artificial neural network (ANN) having thecapability to be trained to perform a currently unknown function. Inmachine learning and cognitive science, ANNs are a family of statisticallearning models inspired by the biological neural networks of animals,and in particular the brain. ANNs can be used to estimate or approximatesystems and functions that depend on a large number of inputs.

ANNs can be embodied as so-called “neuromorphic” systems ofinterconnected processor elements that act as simulated “neurons” andexchange “messages” between each other in the form of electronicsignals. Similar to the so-called “plasticity” of synapticneurotransmitter connections that carry messages between biologicalneurons, the connections in ANNs that carry electronic messages betweensimulated neurons are provided with numeric weights that correspond tothe strength or weakness of a given connection. The weights can beadjusted and tuned based on experience, making ANNs adaptive to inputsand capable of learning. For example, an ANN for handwriting recognitionis defined by a set of input neurons that can be activated by the pixelsof an input image. After being weighted and transformed by a functiondetermined by the network's designer, the activation of these inputneurons are then passed to other downstream neurons, which are oftenreferred to as “hidden” neurons. This process is repeated until anoutput neuron is activated. The activated output neuron determines whichcharacter was read. Similarly, the decision engine 114 can utilize theML model 118 to detect an incident. For example, the decision engine 114can detect, using image recognition techniques, an emergency vehicle inan image captured by the camera 120, can detect, using audio processingtechniques, a siren of an emergency vehicle in a soundwave captured bythe microphone 142, and the like.

At decision block 208, it is determined whether an incident is detectedat block 206. If at decision block 208, it is determined that anincident has not occurred, the decision engine 114 continues to monitorthe sensors 202 for an indication of an incident.

However, if at decision block 208 it is determined that an incident hasoccurred, the control engine 116 initiates recording/storage of datafrom the sensors 202 at block 210. This can include storing previouslycaptured data and/or causing future data to be captured and stored. Thedata can be stored locally, such as in the data store 111, and/orremotely, such as in the data store 151 of the remote processing system150 or another suitable system or device. The control engine 116 canalso take an action at block 212 and/or issue a notification at block214 responsive to the decision engine 114 detecting an incident.Examples of actions that can be taken at block 212 include, but are notlimited to: controlling the vehicle 100 (e.g., causing the vehicle 100to execute a driving maneuver, such as changing lanes, changingvelocities, etc.; causing the vehicle 100 to turn on one or more of itslights; causing the vehicle 100 to roll down/up one or more of itswindows; etc.), causing the recorded data to be modified (e.g.,overlaying GPS data, speed/velocity data, location data, a timestamp,etc. on recorded video; combining recorded soundwaves and recordedvideo; etc.), and other suitable actions. Examples of notifications thatcan be issued at block 214 can include, but are not limited to:presenting an audio and/or visual prompt to an operator or occupant ofthe vehicle 100 (e.g., presenting a warning message on a display withinthe vehicle, playing a warning tone within the vehicle, etc.), alertinga third-party service (e.g., an emergency dispatch service, a knowncontact of an operator or occupant of the vehicle, etc.), sending analert to the other vehicle 154 and/or the remote processing system 150,etc. The type of action taken and/or the type of notification issued canbe based on one or more of: user preferences; type of incident detected;geographic-based laws, regulations, or customs; and the like.

FIG. 3 depicts a flow diagram of a method 300 for automated incidentdetection for vehicles according to one or more embodiments describedherein. The method 300 can be performed by any suitable system or devicesuch as the processing system 110 of FIGS. 1 and 2 , the processingsystem 500 of FIG. 5 , or any other suitable processing system and/orprocessing device (e.g., a processor). The method 300 is now describedwith reference to the elements of FIGS. 1 and/or 2 but is not solimited.

At block 302, the processing system 110 receives first data from asensor (e.g., one or more of the cameras 120-123, 130-133; the radarsensor 140; the LiDAR sensor 141; the microphone 142; etc.) of avehicle.

At block 304, the processing system 110 determines whether an incidentexternal to the vehicle has occurred by processing the first data usinga machine learning model. For example, as described herein, the decisionengine 114 processes the data collected by the sensors. Particularly,the decision engine 114 monitors the sensors 202, using the datareceived/collected at block 204, for an indication of an incident.According to one or more embodiments described herein, the decisionengine 114 can utilize artificial intelligence (e.g., machine learning)to detect features within the sensor data (e.g., a captured image, arecorded soundwave, etc.) that are indicative of an incident. Forexample, using the data received/collected at block 204, the decisionengine 114 can detect the presence of a law enforcement vehicle, amedical support vehicle (such as an ambulance or first respondervehicle), and the like.

At block 306, the processing system 110 initiates recording of seconddata by the sensor responsive to determining that an incident externalto the vehicle has occurred. For example, the processing system 110initiates recording of video responsive to detecting an incident basedon recorded audio.

At block 308, the processing system 110 takes an action to control thevehicle responsive to determining that an incident external to thevehicle has occurred. Taking an action includes the processing system110 causing another system, device, component, etc. to take the action.In some examples, the processing system 110 controls the vehicle 100,such as to execute a driving maneuver (e.g., change lanes, changevelocity, etc.), initiate recording from sensors of the vehicle 100,cause recorded data to be stored in the data store 111 and/or the datastore 151 of the remote processing system 150, and perform othersuitable actions.

Additional processes also may be included, and it should be understoodthat the process depicted in FIG. 3 represents an illustration and thatother processes may be added or existing processes may be removed,modified, or rearranged without departing from the scope and spirit ofthe present disclosure.

FIGS. 4A-4C depict scenarios 400, 401, 402 according to one or moreembodiments described herein. In the scenario 400 of FIG. 4A, thevehicle 100 is shown as having a plurality of microphones 142. Themicrophones 142 record soundwaves emitted by a siren of an emergencyvehicle 410. The data (soundwaves) collected by the microphones 142 canbe processed by the processing system 110 of the vehicle 100 usingspecial triangulation and temporal processing to detect the siren anddetermine a relative position and/or orientation of the emergencyvehicle 410.

In the scenario 401 of FIG. 4B, the emergency vehicle 410 is approachingthe vehicle 100 over time (T) as shown. The vehicle 100 is equipped witha plurality of cameras 420, which may include one or more of the cameras120-123, 130-133 as described herein. The processing system 110 performsa camera-based emergency light flickering detection through temporalintegration to achieve robust relative position and/or orientation ofthe emergency vehicle 410. For example, a frame-based convolutionalneural network (CNN) can be used to perform spatial matching).

In the scenario 402 of FIG. 4C, the emergency vehicle 410 moves from afirst position 411 to a second position 412 over time (T) as shown. Theprocessing system 110 can fuse microphone data (from the microphone 142)and camera data (from the camera 420) using machine learning (e.g.,federated learning). For example, the microphone 142 can provideorientation first and the data from the camera 420 can then be used torefine and provide a range estimation between the vehicle 100 and theemergency vehicle 410 over time.

According to one or more embodiments described herein, datacollection/recording is limited to relevant data and federated learningstatistics, such as emergency vehicles and potential crashed scene toprovide privacy and reduce data usage/bandwidth). According to one ormore embodiments described herein, the collected data can be used todetermine a behavior/intent of a situation (e.g., ignorepassing/traveling emergency vehicle 410 if it is determined that theemergency vehicle 410 is traveling to another location and isunconcerned with the vehicle 100; record/collect data if the emergencyvehicle 410 and the vehicle 100 are stopped together, acceleratingtogether, etc.). In some examples, if a relevant topic is detected, therecorded data can be compressed and sent to a mobile phone, cloudcomputing node, or other device/system (e.g., the remote processingsystem 150) for rendering.

Each of the scenarios of FIGS. 4A-4C depict single-vehicle scenarios.However, multiple vehicle scenarios are also possible. For example,given each single-vehicle processed results, the robustness of theincident detection (e.g., by the decision engine 114) can be improvedbased on observations from multiple vehicles. In such cases, jointprocessing, including 3D triangulation and reconstruction in space andtime, can be processed on the data from the multiple cameras/vehicles.In some cases, information/data from multiple other vehicles couldprovide a cue to the vehicle 100 that an incident is occurring (or hasoccurred), such as by identifying an emergency vehicle using one of theother vehicles. In some examples, the other vehicles can beidentified/selected based on the value of the data they provide. Forexample, an on-coming vehicle can provide a complementary view to thevehicle 100 and more information that may be otherwise unavailable tothe vehicle 100. This is also useful for scene reconstruction. Thisapproach provides improved recorded data (e.g., a 3D view may be able tobe reconstructed because of camera views from multiple vehicles). Insome cases, the recorded data can be compressed and sent to a cloudcomputing environment that can then compose a wholistic-view to berendered on other devices, such as a smart phone. Further, scenereconstruction, including geometry and motion, can be performed based onmultiple views/data from multiple vehicles with only relevant topicsincluded to preserve privacy and reduce data bandwidth.

It is understood that one or more embodiments described herein iscapable of being implemented in conjunction with any other type ofcomputing environment now known or later developed. For example, FIG. 5depicts a block diagram of a processing system 500 for implementing thetechniques described herein. In accordance with one or more embodimentsdescribed herein, the processing system 500 is an example of a cloudcomputing node of a cloud computing environment. Cloud computing cansupplement, support, or replace some or all of the functionality of theelements of the processing system 110, the remote processing system 150,and or the processing system 500.

In examples, processing system 500 has one or more central processingunits (“processors” or “processing resources”) 521 a, 521 b, 521 c, etc.(collectively or generically referred to as processor(s) 521 and/or asprocessing device(s)). In aspects of the present disclosure, eachprocessor 521 can include a reduced instruction set computer (RISC)microprocessor. Processors 521 are coupled to system memory (e.g.,random access memory (RAM) 524) and various other components via asystem bus 533. Read only memory (ROM) 522 is coupled to system bus 533and may include a basic input/output system (BIOS), which controlscertain basic functions of processing system 500.

Further depicted are an input/output (I/O) adapter 527 and a networkadapter 526 coupled to system bus 533. I/O adapter 527 may be a smallcomputer system interface (SCSI) adapter that communicates with a harddisk 523 and/or a storage device 525 or any other similar component. I/Oadapter 527, hard disk 523, and storage device 525 are collectivelyreferred to herein as mass storage 534. Operating system 540 forexecution on processing system 500 may be stored in mass storage 534.The network adapter 526 interconnects system bus 533 with an outsidenetwork 536 enabling processing system 500 to communicate with othersuch systems.

A display (e.g., a display monitor) 535 is connected to system bus 533by display adapter 532, which may include a graphics adapter to improvethe performance of graphics intensive applications and a videocontroller. In one aspect of the present disclosure, adapters 526, 527,and/or 532 may be connected to one or more I/O busses that are connectedto system bus 533 via an intermediate bus bridge (not shown). SuitableI/O buses for connecting peripheral devices such as hard diskcontrollers, network adapters, and graphics adapters typically includecommon protocols, such as the Peripheral Component Interconnect (PCI).Additional input/output devices are shown as connected to system bus 533via user interface adapter 528 and display adapter 532. A keyboard 529,mouse 530, and speaker 531 (or other suitable input and/or outputdevice, such as a touch screen of an infotainment system of a vehicle orthe microphone 142) may be interconnected to system bus 533 via userinterface adapter 528, which may include, for example, a Super I/O chipintegrating multiple device adapters into a single integrated circuit.According to one or more embodiments described herein, one or more ofthe cameras 120-123, 130-133, the radar sensor 140, the LiDAR sensor141, and the microphone 142 is also connected to the system bus 533.

In some aspects of the present disclosure, processing system 500includes a graphics processing unit 537. Graphics processing unit 537 isa specialized electronic circuit designed to manipulate and alter memoryto accelerate the creation of images in a frame buffer intended foroutput to a display. In general, graphics processing unit 537 is veryefficient at manipulating computer graphics and image processing, andhas a highly parallel structure that makes it more effective thangeneral-purpose CPUs for algorithms where processing of large blocks ofdata is done in parallel.

Thus, as configured herein, processing system 500 includes processingcapability in the form of processors 521, storage capability includingsystem memory (e.g., RAM 524), and mass storage 534, input means such askeyboard 529 and mouse 530, and output capability including speaker 531and display 535. In some aspects of the present disclosure, a portion ofsystem memory (e.g., RAM 524) and mass storage 534 collectively storethe operating system 540 to coordinate the functions of the variouscomponents shown in processing system 500.

As used herein, the term module refers to processing circuitry that mayinclude an application specific integrated circuit (ASIC), an electroniccircuit, a processor (shared, dedicated, or group) and memory thatexecutes one or more software or firmware programs, a combinationallogic circuit, and/or other suitable components that provide thedescribed functionality.

The descriptions of the various examples of the present disclosure havebeen presented for purposes of illustration but are not intended to beexhaustive or limited to the embodiments disclosed. Many modificationsand variations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the described techniques.The terminology used herein was chosen to best explain the principles ofthe present techniques, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the techniquesdisclosed herein.

While the above disclosure has been described with reference toexemplary embodiments, it will be understood by those skilled in the artthat various changes may be made and equivalents may be substituted forelements thereof without departing from its scope. In addition, manymodifications may be made to adapt a particular situation or material tothe teachings of the disclosure without departing from the essentialscope thereof. Therefore, it is intended that the present techniques notbe limited to the particular embodiments disclosed, but will include allembodiments falling within the scope of the application.

What is claimed is:
 1. A computer-implemented method comprising:receiving first data from a sensor of a target vehicle; accessing datafrom an other vehicle, determining, by a processing device, whether anincident external to the target vehicle has occurred by processing thefirst data and the data from the other vehicle using a machine learningmodel; responsive to determining that an incident external to the targetvehicle has occurred, initiating recording of second data by the sensor;responsive to determining that an incident external to the targetvehicle has occurred, taking an action to control the target vehicle;reconstructing, at a remote processing system, a three-dimensional sceneof the incident based at least in part on the second data and the dataaccessed from the other vehicle.
 2. The computer-implemented method ofclaim 1, further comprising, responsive to determining that an incidentexternal to the vehicle has not occurred, receiving third data from thesensor of the vehicle, and determining whether an incident external tothe vehicle has occurred by processing the third data using the machinelearning model.
 3. The computer-implemented method of claim 1, whereinthe sensor is a camera.
 4. The computer-implemented method of claim 1,wherein the sensor is a microphone.
 5. The computer-implemented methodof claim 1, wherein receiving the first data from the sensor of thetarget vehicle comprises: receiving audio data from a microphone; andreceiving video data from a camera.
 6. The computer-implemented methodof claim 5, further comprising fusing the audio data and the video data.7. The computer-implemented method of claim 1, further comprising,responsive to initiating recording of the second data by the sensor,overlaying information on the second data.
 8. The computer-implementedmethod of claim 7, wherein the information comprises locationinformation associated with a location of the vehicle, a timestampassociated with a time of the incident, and speed information associatedwith a speed of the target vehicle at the time of the incident.
 9. Thecomputer-implemented method of claim 1, wherein the machine learningmodel is a federated learning model.
 10. The computer-implemented methodof claim 1, further comprising transmitting the second data to a datastore associated with a remote processing system.
 11. Thecomputer-implemented method of claim 1, further comprising responsive todetermining that an incident external to the target vehicle hasoccurred, issuing an alert to an operator of the vehicle.
 12. Thecomputer-implemented method of claim 1, further comprising responsive todetermining that an incident external to the target vehicle hasoccurred, issuing an alert to a third party remote from the targetvehicle.
 13. The computer-implemented method of claim 1 wherein theincident external to the target vehicle includes a law enforcementtraffic stop.
 14. A system comprising: a sensor; a memory comprisingcomputer readable instructions; and a processing device for executingthe computer readable instructions, the computer readable instructionscontrolling the processing device to perform operations comprising:receiving first data from the sensor of a target vehicle; accessing datafrom an other vehicle; determining, by a processing device, whether anincident external to the target vehicle has occurred by processing thefirst data and the data from the other vehicle using a machine learningmodel; responsive to determining that an incident external to the targetvehicle has occurred, initiating recording of second data by the sensor;responsive to determining that an incident external to the targetvehicle has occurred, taking an action to control the target vehicle;and reconstructing, at a remote processing system, a three-dimensionalscene of the incident based at least in part on the second data and thedata accessed from the other vehicle.
 15. The system of claim 14,wherein the sensor is a camera.
 16. The system of claim 14, wherein thesensor is a microphone.
 17. The system of claim 14, wherein receivingthe first data from the sensor of the target vehicle comprises:receiving audio data from a microphone; and receiving video data from acamera.
 18. The system of claim 17, wherein the operations furthercomprise fusing the audio data and the video data.
 19. The system ofclaim 14, wherein the machine learning model is a federated learningmodel.
 20. A computer program product comprising a computer readablestorage medium having program instructions embodied therewith, theprogram instructions executable by a processor to cause the processor toperform operations comprising: receiving first data from a sensor of avehicle; accessing data from an other vehicle; determining, by aprocessing device, whether an incident external to the target vehiclehas occurred by processing the first data and the data from the othervehicle using a machine learning model; responsive to determining thatan incident external to the target vehicle has occurred, initiatingrecording of second data by the sensor; responsive to determining thatan incident external to the target vehicle has occurred, taking anaction to control the vehicle; and reconstructing, at a remoteprocessing system, a three-dimensional scene of the incident based atleast in part on the second data and the data accessed from the othervehicle.